Redundant Error Detection in a Clinical Diagnostic Analyzer

ABSTRACT

Disclosed is a clinical diagnostic analyzer employing a redundant error detection capability to further examine the internal message traffic for possible errors or questionable results by comparing the actual parameters against a fingerprint generated for each assay using an assay database and a configuration file. This testing does not rely on inputs from the software module being tested, and hence is an independent test. Further, a testing mechanism is provided to test the Redundant Error Detection (“RED”) capability itself.

BACKGROUND

Repairing or correcting degraded assay performance in clinical analyzersystems has proven to be more difficult than correcting electrical ormechanical issues. Estimating the expected performance and monitoringthe actual performance of clinical laboratory testing and diagnosticsystems requires detailed and current information. Design andmanufacturing clinical diagnostic systems requires testing andintegration of thousands of components and subsystems in a complexsoftware coordinated combination for carrying out accurate assays whilemaintaining a high throughput. An example is provided by the VITROS 4,3™and ECi line of clinical Diagnostic Analyzers manufactured by OrthoClinical Diagnostics (“OCD”) of Raritan, N.J.

A clinical diagnostic analyzer contains extensive software monitoringand controlling its operation. Testing and examining every possiblestate of the analyzer is often not possible in view of the large numberof states possible for the combination of software and hardware over theexpected life and use of the analyzer. Testing the software in aclinical diagnostic analyzer is expensive, time consuming and difficult.For instance, if a subsystem in the analyzer fails, it is reduced, whichis declared to be inactive, and needs to be restarted and initializedbefore use. Asynchronous messages, generated, for instance due to anunexpected opening of the cover of the incubation chamber, may requirethe affected replicates to be treated as potentially tampered with.Further, recovery from error conditions may affect multiple other testsdue to exceeded limits, missed steps and many other undetected errors.Errors may also cause additional errors to ripple through multiple othertests. Further, such errors or their effect may not be detected in timeby the routine process for controlling and regulating the operation of aclinical diagnostic analyzer because different tests may be affected todifferent degrees. The alternative often is to discard valuable patientsamples in various stages of processing and restart the entireinstrument, which results in reduced throughput and efficiency.

One of the goals in designing clinical diagnostic analyzers is toprevent the release of questionable laboratory results in view of theharm that may result to a patient from treatment or lack thereof basedon erroneous results. Thus, timely error detection is of considerableimportance in clinical diagnostic analyzers. Indeed, it is desirable togo beyond ensuring assay and equipment performance to merely satisfyhealth and safety guidelines in order to reduce the cost of assays whileimproving the reliability of assay results. At the same time, it isimperative to not discard valid results based on an over-inclusivecriterion.

Laboratories use a wide range of techniques to help prevent incorrect oreven potentially incorrect results from being released. These includebut are not limited to assay control charts, equipment maintenanceprocesses/logs, periodic analyzer maintenance, comparison to patientnormal performance for the particular assay, reviewing changes sincelast result for a patient, grouping of results to calculate aphysiologically stable quantity. All of these approaches are designed toreact to a performance issue after results are released. Releasing aresult typically means not only providing the result, but alsoindicating reliability of the result. Many checks are performed asinfrequently as once a day and are suitable only for detecting long-termtrends or frequent outliers or highly degraded performance. This putsthe laboratory and patient at risk due to release of assay data withdegraded performance for hours or even days before it is detected usingthese methods. In a busy lab, this can result in hundreds of patientsamples being retested with a heightened risk of other than the desiredtreatment while laboratory throughput decreases.

Clinical diagnostic analyzers, like other complex systems comprisingscheduler like controllers, control sub-systems using message trafficcomprising commands, responses and asynchronous messages, interrupts andthe like. Typically, the scheduler, usually implemented as a softwaremodule, controls the operation of the clinical diagnostic analyzer byallocating resources, scheduling the desired tests and tracking theperformance of the various commands. The scheduler issues commands tovarious subsystems, which send back a response indicating the result ofcarrying out the command. The subsystems may, in turn, in response toreceiving a command from the scheduler, send one or more commands toother software modules and receive messages from them indicating theresult of executing the corresponding command.

All Subsystem Commands, Command Responses, and Asynchronous messages maybe considered to be a sub-system input/output. The software module, asubsystem manager, writes these messages into Emulator Files. TheEmulator Files, while voluminous, may be reviewed in course oftrouble-shooting. This task, however, tends to be slow and impracticalto flag and prevent questionable results from being released withoutinordinately delaying the release of results. Notably, clinicaldiagnostic analyzers typically support testing of STAT samples requiringexpedited processing. Some additional strategies to track or estimatethe performance of complex systems that are also not suitable forhandling clinical diagnostic analyzers are describe next.

U.S. Pat. No. 7,254,601 (the “'601 patent”) discloses a method formanaging remotely deployed intelligent equipment, but, does not teachtimely prevention of release of questionable laboratory results.

U.S. Pat. No. 6,757,714 (the “'714 patent”) and patents and patentapplications related thereto disclose obtaining and communicating anerror condition of an apparatus via email to a remote server. Apredefined template is used to generate the e-mail message by obtainingand inserting one or more variables into the template. The errorcondition may be included as part of a body of the e-mail message or aspart of an attachment to the e-mail message. The error condition isreported using a self-describing computer language, such as eXtensibleMarkup Language (XML). In general, the error condition is determinedwith the aid of an embedded controller. The remote server passes theerror condition to a customer relationship management system. The '714patent does not address avoiding the release of erroneous results bypreventing the machine from releasing questionable results due tounexpected scheduler or software errors.

What is needed is a reduction in the need to exhaustively review andtest every possible state of a clinical diagnostic analyzer in order toensure robust performance. Instead, better error detection strategiesare needed to ensure reliability of results released by a laboratory.

SUMMARY

A preferred embodiment provides a system and method for detecting orpredicting errors to prevent release of questionable laboratory testresults. Briefly, the preferred embodiment employs a redundant errordetection capability to further examine the message traffic for possibleerrors or questionable results by comparing the actual parametersagainst an expected fingerprint generated for each assay using an assaydatabase and a configuration file or specification. This testing doesnot rely on inputs from the software module being tested, and hence isan independent test. For instance, if the scheduler software performs inunexpected ways, it is desirable to have an independent method to flagsuch errors before any results affected by such errors or issues arereleased. Further, a testing mechanism is provided to test the RedundantError Detection (“RED”) capability itself.

The fingerprint is generated, preferably dynamically, from (i) an assaydatabase capable of providing protocol information for each assay, and(ii) hard-coded information in the form of a configuration file for theassay, replicate or device of interest. The fingerprint is a set ofexpected events required for successful completion of the test inquestion. At the same time, the number of events in a fingerprinttypically is less than the exhaustive listing of events that areactually executed for the test. The reduction in the number of events inthe protocol database to generate the fingerprint is guided by theconfiguration specification. Preferably, the configuration specificationis merged with information from the assay database to generate theexpected fingerprint.

With a generated fingerprint, RED reviews the message traffic andidentifies discrepancies in the form of events required by thefingerprint that are absent from the message traffic to flagquestionable results. RED can also review emulator files or data loggingfiles to gain access to the message data and identify discrepanciesafter results have been processed. This ability is useful in validatingand testing software including validating newer versions of RED itselfin an economical manner.

In another aspect, RED is integrated into existing analyzers easilybecause it requires few modifications and leaves the subsysteminput-output traffic largely unchanged.

A preferred embodiment comprises a module executable on a processor in aclinical diagnostic analyzer for detecting questionable laboratoryresults in the clinical diagnostic analyzer. The module includes amechanism for receiving a copy of messages comprising SubsystemCommands, Command Responses, and Asynchronous messages. The module alsoincludes a fingerprint of expected events to be tested for a replicate.The fingerprint is generated, in response to a preview replicatecommand, from at least a configuration file and an assay database,wherein the assay database contains details for an assay required by thereplicate. The module includes instructions for completing a comparison,in response to a replicate check command, between the received messagesand the fingerprint. The module includes instructions for reporting aresult of the comparison; and releases resources required for theredundant checking of the replicate, in response to receiving areplicate complete command.

J In the preferred embodiment, the RED functionality may be readilyturned off by an operator dynamically. This may be useful if an error inRED is suspected. If a failure or questionable results are detected, theRED functionality logs corresponding conditions. Such logging notifiesthe operator of the detected condition and is useful in debugging andimproving software. However, the decision to not release results may beturned off until suspected problems in RED itself are resolved.

This disclosure includes a method for testing a software module in aclinical diagnostic analyzer. The method comprises the steps ofgenerating subsystem commands, command responses, and asynchronousmessages suitable for delivery to a redundant error detector. Theredundant error detector reviews the generated subsystem commands,command responses, and asynchronous messages for inconsistencies. In theevent such inconsistencies are identified, the software module isdebugged to ensure reliable and consistent performance. Notably, thesoftware module being tested could be the redundant error detectorsoftware. Further, part of or an entire system comprising softwaremodules may be tested in this manner.

The subsystem commands, command responses, and asynchronous messages areadvantageously generated from a script file containing a record of oneor more prior runs of the analyzer. Then the performance of the clinicaldiagnostic analyzer with alternative versions of the software can becompared using the redundant error detector to detect inconsistencies inany of the required events in a fingerprint. The script file can includecommands in addition to data from the example operation of the clinicaldiagnostic analyzer.

In another aspect, a preferred embodiment is a method for preventingrelease of a questionable result by a clinical diagnostic analyzer. Themethod comprises facilitating the steps of receiving a preview replicatecommand for a replicate; allocating resources for redundant errorchecking for the replicate; generating a fingerprint of required eventsfor the replicate using at least a configuration file and an assaydatabase, wherein the fingerprint has fewer events than the events thatcould be generated from the assay database for the replicate; comparingmessages corresponding to processing out the replicate with thefingerprint; sending a result of the comparison to prevent the releaseof a result corresponding to the replicate by the clinical diagnosticanalyzer; and logging conditions corresponding to the replicate. Itshould be noted that allocation and release of resources required forredundant error checking need not be express and instead may be handledby the operating system or other functionality automatically dependingon the system or the underlying programming language. Advantageously,the configuration file is coded in eXtensible Markup Language forimplementation flexibility, which makes it readable by humans and easyto send as text to remote analyzers.

These and other features in some preferred embodiments are describedbelow in greater detail with the aid of illustrative figures. These andother features in some preferred embodiments are described below ingreater detail with the aid of illustrative figures and modules orsubmodules. In a preferred embodiment, a module (and a sub-module)reflects functional partitioning. A module or sub-module may use sharedresources to perform its function. Resources typically are processortime slices, database data, data in memory shared with other modules,sub modules, shared library functions and the like as is well known toone having ordinary skill in the art. A developer, having ordinary skillin the art, knows how to create a module. Mapping the functionalpartitioning into executable code is familiar to one having ordinaryskill in the art. The figures are briefly described next.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a schematic of the processing modules in a clinicaldiagnostic analyzer coupled to a redundant error detector.

FIG. 2 illustrates a schematic of a software testing arrangement using aredundant error detector.

FIG. 3 illustrates the redundant error detection functionality andvarious components within it or closely interacting with it, which mayfurther integrated.

FIG. 4 illustrates a method for checking for errors using a redundanterror detector.

FIG. 5 illustrates a method for evaluating and debugging software usinga redundant error detector.

FIG. 6 illustrates the redundant error detection functionality andvarious components within it or closely interacting with it, which maybe further integrated.

FIG. 7 illustrates an example redundant error detection functionalitychecking a message stream for device and replicate events while updatingdevice state maintained by the redundant error detection functionality.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

A ‘basic event’ in a clinical diagnostic analyzer, typically, is thedetection of one or more parameters or a change therein. The parametersare values or descriptors of a state of the clinical diagnosticanalyzer. An event occurs upon fulfillment of one or more specifiedconditions. A large number of such basic events are typically predefinedfor clinical diagnostic analyzers. The message traffic comprisingcommands, responses and asynchronous messages is an example of a subsetof events of interest.

Redundant Error Detector (“RED”) module, a software feature, detectsdiscrepancies between sequences of expected events and actually observedevents in the course of operating an analyzer or another sophisticatedprocessing system. Sequences of expected events are formulated toprovide a fingerprint for each test type run to avoid repeatedexhaustive review of all events. RED provides not only softwareverification but also regression testing of replicate runs. In thiscontext, an event denotes detection of one or more parameters or achange therein. Some examples of events include issuing of or receivinga command, making a measurement, issuing or receiving a response toconfirm execution of a command, an interrupt and the like.

RED is compatible with OCD's clinical diagnostic analyzer VITROS 5600™and VITROS 3600™, which provide a preferred platform for implementingRED. RED is expected to be useful to other clinical diagnostic analyzersas well. Each of VITROS 5600™ and VITROS 3600™ use functions provided bya Master Scheduler, which can support multiple and different types ofchemistry platforms. OCD's Master Scheduler can also control a roboticarm and pipettors shared between the various platforms to provide randomaccess to input patient samples, which improves throughput. The randomaccess to the input samples substantially eliminates the typicalrequirement to process samples in the order they are input.

OCD's Master Scheduler performs scheduling functions, for instance, toallocate resources to samples input in any order. Other not so versatileschedulers also benefit from the addition of the RED functionality.OCD's Master Scheduler determines a schedule to minimize the Time toComplete for the tests as a whole thereby maintaining high throughputwhile honoring applicable protocol restrictions.

The benefits of OCD's Master Scheduler include the synergistic effectsof two-dimensional random access to the input samples while providingaccess to resources including, for example, multiple platforms, supplyof consumables including thin film slides, reaction vessels andcuvettes, along with a plurality of sensiometric devices such aselectrometers, reflectometers, luminescence, light transmissivity,photon detection, an incubator for heating the samples, a supply ofreagents, and a plurality of reagent delivery subsystems, all of whichcan be accessed and used readily.

This functionality naturally entails additional complexity, which, inturn, requires testing to ensure reliability. RED provides additionalassurances against the release of questionable results. Typically, amanufacturer of a clinical diagnostic analyzer provides a master list ofdetails of supported assay protocols with its analyzers. This masterlist of protocol details may be edited to various degrees to add ormodify the supported tests. The protocol details are stored in an assaydatabase, or acquired from another information source, which isconsulted for each supported assay by a preprocessing module to generatethe parameters and resources required by a test. Although protocoldetails may not be in the form of events expected on the correspondinganalyzer, such events can be generated using knowledge of a particularanalyzer. The Scheduler uses the protocol details to allocate resourcesand schedule various steps to improve throughput and ensure quality.

To run each test, the Scheduler module issues multiple commands to thesubsystem modules. When a command is carried out, various measurementsare made, gears may turn or other operations take place and a responsemessage is sent back to the scheduler to confirm the command was carriedout successfully to complete the loop. If a subsystem detectedunsuccessful execution of the command, then the corresponding responsesent to the Scheduler typically includes this information.

Nevertheless, undetected errors in the operation of clinical analyzersare an expensive and undesirable reality. For instance, if a sub-systemfails, it is ‘reduced,’ that is made unavailable and all replicatesrequiring the use of the reduced sub-system stall. However, other testscontinue to progress to best use the analyzer resources. To ensurereliability, the reduced sub-system is restarted prior to processingsamples, which procedure makes the sub-system performance predictable,but may cause unanticipated delays and perturbations in the scheduling.Similar uncertainty may result due to unplanned events, such as anoperator opening the cover of an incubation chamber and potentiallycontaminating all samples contained therein. Scheduler software may haveanomalies resulting in a particular step not being followed. If thefailure to perform the step is a rare occurrence, it may not be detectedreadily during testing. Some such errors in the Scheduler software maybe exacerbated by unplanned events such as reduction of varioussub-systems, initialization of these sub-systems, and otherinterruptions.

A preferred embodiment provides a system and method for detecting orpredicting errors to prevent release of questionable laboratory testresults. A list of required events, as opposed to all possible events,for each supported test is prepared in advance in the form of aconfiguration file. This configuration specification is preferablydefined in an eXtensible Markup Language (“XML”) format.

There are many advantages in using XML. XML is useful for definingevents, including those that are combinations or variations of basicevents. XML encoded instructions and descriptions can be implemented anddebugged rapidly in real time using an interpreter. Further, XML baseddefinitions are readily communicated by treating them as text. However,XML tags, although readable by humans, often describe very differentfunctions and data using similar or even the same tag names.Nevertheless, in a context XML provides a description of the functionsperformed by executing XML compliant instructions. Therefore, given aproper context, XML provides a convenient method for communicating data,instructions and combinations thereof.

In effect, a specific instance of XML designed to facilitate aparticular set of tasks at a target creates a “new machine” capable ofperforming particular functions. Thus, a circuit performing function(s)specified by one or more XML tags in response to interpreting XMLencoded instructions is a structure performing the indicatedfunction(s).

An example event described using XML in a configuration specification isshown below:

 <?xml version=“1.0” ?>  − <!-- $Id: RED_MsPmRepSeq.xml,v 1.8 2008/03/3114:04:05 mreed7 Exp $   -->  − <RED_STEP RED_STEP_NAME=“RED_MsPmRepSeq”RED_STEP_DESC=“MicroSlide PM Replicate”>  − <RED_EVENTRED_EVENT_NAME=“Aspirate ERF command” RED_EVENT_DESC=“Aspirate ERFcommand”>  − <RED_CRITERIA_SET>   <RED_CRITERIARED_EVENT_TOKEN_NAME=“EVENT_TYPE” RED_CRITERIA_VALUE=“CMD” />  <RED_CRITERIA RED_EVENT_TOKEN_NAME=“DEV”RED_CRITERIA_VALUE=“DRY_REF_MET_PUMP_DEV” />   <RED_CRITERIARED_EVENT_TOKEN_NAME=“CMD” RED_CRITERIA_VALUE=“ASPIRATE” />  </RED_CRITERIA_SET>  − <RED_ACTIONRED_ACTION_NAME=“RED_SET_SEQ_START_TIME_IF_NOT_SET”RED_ACTION_COMMENT=“Set the start time of the sequence to equal theevent time plus the offset” RED_EVENT_TOKEN_NAME=“TIME”RED_ERROR_ID=“1”>  − <RED_LIST RED_LIST_TYPE_NAME=“PRIM”SC_SCRIPT_DATA_NAME=“RED_EVENT_MS_TIME_INTERNAL”>   <RED_VALUERED_VALUE_TYPE_NAME=“RED_STATIC_VALUE” RED_VALUE_VALUE=“−1800” />  </RED_LIST>   </RED_ACTION>  − <RED_ACTIONRED_ACTION_NAME=“RED_CHECK_EVENT_EQ_CFG_TIME_RANGE”RED_ACTION_COMMENT=“Check that the event starts within the specifiedtime range from the start of the step.” RED_EVENT_TOKEN_NAME=“TIME”RED_ERROR_ID=“2”>  − <RED_LIST RED_LIST_TYPE_NAME=“PRIM”SC_SCRIPT_DATA_NAME=“RED_EVENT_MS_TIME_INTERNAL”>   <RED_VALUERED_VALUE_TYPE_NAME=“RED_PREVIEW_MSEC_TIME_RANGE_VALUE”RED_VALUE_VALUE=“0” />   <RED_VALUERED_VALUE_TYPE_NAME=“RED_PREVIEW_MSEC_TIME_RANGE_VALUE”RED_VALUE_VALUE=“6550” />   </RED_LIST>   </RED_ACTION>  − <RED_ACTIONRED_ACTION_NAME=“RED_CHECK_DEV_EQ_CFG_PARAM” RED_ACTION_COMMENT=“RefMetCAM positioned for aspirate” RED_DEVICE_NAME=“RED_MS_REF_MET_DEV”RED_ERROR_ID=“7”>  − <RED_LIST RED_LIST_TYPE_NAME=“PRIM”SC_SCRIPT_DATA_NAME=“REF_MET_CAM_POS”>   <RED_VALUERED_VALUE_TYPE_NAME=“RED_STATIC_VALUE” RED_VALUE_VALUE=“RESERVOIR” />  </RED_LIST>  − <RED_FAIL_CMDS>   <RED_TEST_CMDRED_TEST_CMD_NUMBER=“69” />   </RED_FAIL_CMDS>   </RED_ACTION>  −<RED_ACTION RED_ACTION_NAME=“RED_CHECK_EVENT_EQ_DEV_PARAM_BY_TIME_RANGE”RED_ACTION_COMMENT=“Check that RefMet has been used in last 15 Minutes”RED_DEVICE_NAME=“RED_MS_REF_MET_DEV” RED_ERROR_ID=“8”>  − <RED_LISTRED_LIST_TYPE_NAME=“PRIM” SC_SCRIPT_DATA_NAME=“TIME_INTERNAL”>  <RED_VALUE RED_VALUE_TYPE_NAME=“RED_STATIC_VALUE” RED_VALUE_VALUE=“{ 00 }” />   <RED_VALUE RED_VALUE_TYPE_NAME=“RED_STATIC_VALUE”RED_VALUE_VALUE=“{ 900 0 }” />   </RED_LIST>   </RED_ACTION>  </RED_EVENT>   </RED_STEP>

In the above event specification, one event is described within a “REDSTEP” tag, although multiple events are typically describe inconfiguration specifications, which may be in the form of configurationfiles. A set of criteria describing tags provide background andcontextual information followed by multiple “RED ACTION” tag sets,within each of which may be nested “RED LIST” tags, within each of whichmay be listed “RED VALUE” tags describing individual requirementsdefining conditions for satisfying the event. All events so defined mustbe satisfied in the preferred embodiment by the message streamcorresponding to a test of interest for the results of the test to bevalidated. Further, parameters to be passed in the event of failure orsuccess may also be included along with comments and other commands orinstructions. As will be apparent, this is not the only scheme forspecifying events in a fingerprint. Alternative tag sets may be devisedwith no loss of generality or the interpretation of particular tagsmodified without departing from the sprit of the disclosed preferredembodiment.

Accordingly, at runtime a more complete description of required events,a fingerprint, for each test is generated using the assay database incombination with the configuration specification. This approach not onlyconserves required memory resources, but also provides flexibilitybecause updates or changes to the assay database or the configurationspecification are automatically reflected in the generated fingerprint.

As shown in FIG. 1, Scheduler 100 communicates with various subsystemsincluding Scripting 105, which in turn, has a two-way communicationswith the Subsystem Manager 110. Subsystem Manager 110 issues a commandto Subsystem 115 and receives a response confirming execution of thecommand as is shown. The Subsystem Manager 110 typically sends themessage traffic comprising commands and responses as well asasynchronous messages to an Emulator File 160 and to RED Buffer 120,which is a mechanism for transferring messages and information toRedundant Error Detector 130. A preferred RED Buffer 120 is implementedas a queue.

Scheduler 100 also sends commands to Postprocessor 155 by sendingcommands to a Postprocessing Buffer 150. Postprocessor 155 sendscommands to a RED Buffer 120. RED Buffer 120 also receives a copy of themessage traffic being recorded in Emulator File 160. Thus, RedundantError Detector 130 receives as input the message traffic comprisingcommands and responses as well as asynchronous messages and commandsfrom Postprocessor 155. Further, Redundant Error Detector 130 has accessto Configuration File 145.

Control 180 controls Scheduler 100 through a two-way linkage. Similarlinkages connect Sample Handler 170 to Control 180 and the SubsystemManager 110. Under the direction of Control 180, required testinstructions for an input set of patient samples are converted by apreprocessor functionality, using Assay Database 140, to commands forScheduler 100. Scheduler 100 then allocates resources and issuescommands to carryout the instructions. As noted, the expected executionorder of events by Scheduler 100 may not always hold true.

FIG. 3 illustrates the role of Redundant Error Detector 130 in providingan independent test on the results to identify questionable resultswithout increasing the complexity of Scheduler 100 or other components.At the start of a replicate, during step 300, Postprocessor 155 sends aPreview Replicate command to RED Buffer 120. In response to thiscommand, Redundant Error Detector 130 allocates resources during step310 and generates a fingerprint by using Configuration File 145 andAssay Database 140 to determine the parameters and nature of events tobe checked for in the message traffic, also received at RED Buffer 120.

The fingerprint is generated, preferably dynamically, from (i) an assaydatabase capable of providing protocol information for each assay orreplicate, and (ii) hard-coded information in the form of aconfiguration file for the assay, replicate or device of interest. Thefingerprint is a set of expected events required for successfulcompletion of the test in question. At the same time, the number ofevents in a fingerprint typically is less than the exhaustive listing ofevents required to execute the test. Configuration File 145 defines alist of expected events for a given step in a protocol. The reduction inevents reduces the effort required of RED as well as the complexity ofRED.

A preferred fingerprint includes critical events, whose performance isnecessary to generate a reliable result and events that improve theaccuracy of the results. Critical events' successful completion mayitself indicate successful completion of many events that are notexpressly checked as part of the fingerprint.

In the preferred embodiment, parameters configurable per assay are readfrom the Assay Database 140 at run-time. In this preferred embodiment,the expected events of MicroSlide and SI replicates (run on two types ofplatforms) are completely defined by a single configuration file inConfiguration File 145 while the expected events corresponding to aMicroWell and MicroTip replicate (run on two additional types ofplatforms) are defined by merging multiple configuration files together.In this preferred embodiment, each configuration file defines oneprotocol step and Assay Database 140 is queried to determine theprotocol steps required to run each replicate. At runtime RedundantError Detector 130 generates the list of expected events by mergingConfiguration Files 145 corresponding to each protocol step of the givenassay type. Each command event within a replicate step contains anaction that causes Redundant Error Detector 130 to check that the eventoccurred within an allowed time range. At runtime (as the steps aremerged together) the allowed time ranges are modified based upon stepdurations and incubation durations.

Redundant Error Detector 130 preferably processes each event for alldevices prior to processing the event for all pending replicates. Eachevent received by RED is compared to each expected event within eachDevice Sequence in the order in which the expected events are defined.If an event matches one set of criteria defined for that expected eventthen the actions defined within that expected event are executed. Once areceived event matches one expected event within a Device Sequence(which is the definition of the events, criteria, actions, andparameters that correspond to one device) no other expected eventswithin that Device Sequence will be checked or executed until the nextevent is received from RED Buffer 120.

FIG. 6 further describes one such exemplary preferred method for testingusing RED. During step 605 an event is selected from a generatedfingerprint. This is followed by the selection of a Device Eventcorresponding to a selected fingerprint event during step 610. DeviceEvents are preferably specified in the configuration specification.They, however, are not merged with the assay protocol information as isthe case with generating a fingerprint. Instead, they are used toindependently evaluate device state. A Device Event verifies the stateof a device at the time an event of interest takes place.

For instance, if a reading is obtained from an optical reader in aselected event, then a corresponding device event may be to check if theoptical reader was tested against a control within a prescribed periodof time prior to the test reading, and possibly an event after the testreading was taken to ensure the optical reader was fully functional.Failure in one or both of these device events, checked by comparing tothe device state immediately preceding and following the test readingmay mark a replicate as suspect. Thus, failure in a Device Event,checked by comparing to the device state immediately preceding andfollowing the test reading, may identify a suspect replicate.

Control flows from step 610 to decision step 615 during which theselected device event is tested in view of a message in the messagestream being monitored to ensure it is satisfied. If the event is notsatisfied then control flows to step 620 to mark the replicate assuspect and the method concludes. Other embodiments may allow the methodto continue to execute with no loss of generality. Alternatively, if theevent is satisfied, control flows to step 625 during which the need toevaluate another device event is evaluated. If there is at least oneadditional device event at this stage, control flows to step 630 toselect another device event and control returns to step 610.

Alternatively, if there are no further device events corresponding tothe fingerprint event selected in step 605, the selected event isevaluated in step 635. If the selected event is satisfied, then, controlflows to step 640 to determine if there are additional events in thefingerprint. Another event is selected, then, during step 645, ifpossible, and control returns to step 610. Else, control flows to step650 to mark successful matching of the fingerprint with the messagestream or event data being examined. The method then concludes. As isreadily appreciated, this method is suitable for analyzing a messagestream—including message streams not analyzed in real-time or those thatare generated by a simulation.

Expected events may differ between platforms and analyzers. Preferredimplementation of Redundant Error Detector 130 primarily evaluates twotypes of events: (i) Replicate Events; and (ii) Device Events. This isillustrated in the preferred exemplary embodiment illustrated in FIG. 7.

During step 700 a message in received as part of the monitored messagetraffic. During step 705, the message is used to update a description ofDevice State information, if applicable, maintained by RED. In apreferred embodiment, RED maintains state for each device of interest,which states are based on the message traffic monitored by RED. Itshould be noted that not all messages resulting in updating of devicestate may be part of the relevant fingerprint.

The received message is next employed to test for device events. ADevice Event is selected during step 710. Device Events are preferablyspecified in the configuration specification. They, however, are notmerged with the assay protocol information as is the case withgenerating a fingerprint. Instead, they are used to independentlyevaluate device state. A Device Event verifies the state of a device atthe time an event of interest takes place. Failure in a Device Event,checked by comparing to the device state immediately preceding andfollowing the test reading, may identify a suspect replicate.

Control flows from step 710 to decision step 715 during which theselected device event is tested in view of the message to ensure it issatisfied. If the event is not satisfied then control flows to step 720to mark the replicate as suspect. Alternatively, if the event issatisfied, control flows to step 725 during which the need to evaluateanother device event is evaluated. If there is at least one additionaldevice event at this stage, control flows to step 730 to select anotherdevice event and control returns to step 715.

Alternatively, if there are no further device events, an event isselected from the corresponding fingerprint during step 735. Theselected event is evaluated in step 740 in view of the message. If theselected event is satisfied, then, control flows to step 745 todetermine if there are additional events in the fingerprint. Anotherevent is selected, then, during step 750, if possible, and controlreturns to step 740. Else, control flows to step 755 to mark successfulmatching of the fingerprint with the message stream or event data beingexamined. The method then concludes. As is readily appreciated, thismethod is suitable for analyzing a message stream—including messagestreams not analyzed in real-time or those that are generated by asimulation.

Replicate Events

Some examples of Replicate Events in a preferred embodiment are:

MicroSlide PM Replicate is a set of critical events required to processa Potentimetric MicroSlide replicate An example MicroSlide PM Replicatepreferably includes multiple events such as:

Event # 1: Aspirate command;

Event # 2: Aspirate response;

Event # 3: Plunge Slide Supply Cart command;

Event # 4: Plunge Slide Supply Cart response;

Event # 5: Dispense slide from cart command;

Event # 6: Dispense slide from cart response;

Event # 7: Dispense sample to slide command;

Event # 8: Dispense sample to slide response;

Event # 9: Dispense to slide command;

Event # 10: Dispense to slide response;

Event # 11: Push slide to Ring command;

Event # 12: Push slide to Ring response;

Event # 13: Electrometer Read command; and

Event # 14: Electrometer Read response.

Each event requires some conditions to be met as described by specifiedparameters. For instance, Event # 11, Push slide to PM Ring command,requires testing that the event starts within the specified time rangefrom the start of the step. Other events, such as Event # 10 requirejust ‘success’ to be tested to pass. Additional non-exhaustive examplesof Replicate Events include

MicroSlide CM Replicate Incubating in the Outer Ring is a set ofcritical events critical events required to process a Colormetric orRate MicroSlide replicate that performs is major incubation periodwithin the outer ring of the CM Rate Incubator.

MicroSlide CM Replicate Incubating in the Inner Ring is a set ofcritical events required to process a Colormetric or Rate MicroSlidereplicate that performs is major incubation period within the inner ringof the CM Rate Incubator.

MicroSlide IR Replicate is a set of critical events required to processa ImmunoRate MicroSlide replicate.

MicroTip Reagent Addition Step is a set of critical events required toprocess the step where reagent is added to a cuvette within a MicroTipreplicate.

MicroTip Sample Addition Step is a set of critical events required toprocess the step where neat sample is added to a cuvette within aMicroTip replicate.

MicroTip Diluted Sample Addition Step is a set of critical eventsrequired to process the step where diluted sample is added to a cuvettewithin a MicroTip replicate.

MicroTip Read Step is a set of critical events required to process thestep where the cuvette is read using the Photometer within a MicroTipreplicate.

MicroWell 1st Well Pretreatment Step is a set of critical eventsrequired to process the step where neat sample is added to apretreatment well within a MicroWell replicate.

MicroWell Not 1st Well Pretreatment Step is a set of critical eventsrequired to process the step where diluted sample is added to apretreatment well within a MicroWell replicate.

MicroWell 1st Well Dilution Step is a set of critical events required toprocess the step where neat sample is added to a dilution well within aMicroWell replicate.

MicroWell Not 1st Well Pretreatment Step is a set of critical eventsrequired to process the step where diluted sample is added to a dilutionwell within a MicroWell replicate.

MicroWell Reagent Addition Step To Middle Ring Well is a set of criticalevents required to process the step where reagent is added to apretreatment well or a dilution well within a MicroWell replicate.

MicroWell Reagent Addition Step To Outer Ring Well is a set of criticalevents required to process the step where reaction well within aMicroWell replicate.

MicroWell 1st Well Sample Step is a set of critical events required toprocess the step where neat sample is added to a reaction well within aMicroWell replicate.

MicroWell Not 1st Well Sample Step is a set of critical events requiredto process the step where diluted or pretreated sample is added to areaction well within a MicroWell replicate.

MicroWell Preliminary Wash Step is a set of critical events required toprocess the step where a reaction well receives its preliminary wellwash within a MicroWell replicate.

MicroWell Final Wash Step is a set of critical events required toprocess the step where a reaction well receives its final well wash andis read using the Luminometer within a MicroWell replicate.

Sample Integrity Step is a set of critical events required to processSample Integrity readings of a sample.

Device Events

Some preferred examples of device events include those for MicroSlideElectrometer. Independent of the sample slide, the electrometer reads areference slide to ensure at least one successful read of a referenceslide within a prescribed time period. Specifying multiple criticalevents included in a fingerprint captures this requirement. Otherdevices with critical events specified may include incubation rings orareas, conveyor or ring transport mechanisms, shuttles, meteringdevices, controls on metering, reflectometers, slide supplies, bufferrings, movement arms, STAT lanes, reagent supplies, washing stations,blade inserters, blade dumpers, photometers and the like. This list isnot exhaustive.

With a generated fingerprint, Redundant Error Detector 130 reviews themessage traffic and identifies discrepancies in the form of eventsrequired by the fingerprint that are absent from the message traffic.For each received message, RED preferably matches Replicate and DeviceEvents in a fingerprint as illustrated in FIG. 7 to perform associatedactions.

In step 340 of FIG. 3, in response to receiving a Check Replicatecommand in step 330, Redundant Error Detector 130 completes thediscrepancy identification survey by reviewing messages with a timestamp between that of the Preview Replicate command and the CheckReplicate command. Results corresponding to such discrepant events areflagged as questionable.

During step 350, Redundant Error Detector 130 sends the result ofidentifying discrepancies to Postprocessing Buffer 150, which is amechanism for transferring messages and information to Postprocessor155. If the result is determined to be questionable in step 360, then,Redundant Error Detector 130 also identifies the test result as suspectwith logging of the corresponding conditions during step 370. Finally,during step 380 a Complete Replicate command is received. Questionableresults are preferably not released by Postprocessor 155, and naturallyby the clinical diagnostic analyzer. If needed, sequestered resourcesare released during step 390.

Redundant Error Detector can also review emulator files or data loggingfiles to gain access to the message data and identify discrepancies in anon-real-time mode. This ability is useful in validating and testingsoftware including validating newer versions of RED itself in aneconomical manner. FIG. 2 shows a preferred configuration for suchtesting while FIG. 4 illustrates some of the steps in the preferredembodiment.

During step 400 of FIG. 4, a script file, RED Script File 200 is used byRedundant Error Detector Driver 210 to generate ordered entries for REDBuffer 220. RED Script File 200 is a record of messages in an exampleoperation of the clinical diagnostic analyzer coupled with some commandstypically to assist in the testing. Redundant Error Detector 230processes RED Buffer 220 to match a dynamically generated fingerprintusing configuration files and Assay Database 240. The result of theprocessing by Redundant Error Detector 230 is sent to PostProc Buffer250, which in turn leads back to Redundant Error Detector Driver 210,which evaluates the performance of the test software—and may lead todebugging the same. The test software preferably is Redundant ErrorDetector 230. Alternatively, the test software may be a module used inthe example operation to generate data reflected in RED Script File 200.The data reflected in RED Script File 200 in alternative embodiments isfrom an emulator file generated as depicted in FIG. 1 or even adatalogger file being mined to identify errors in an analyzer underinvestigation. Redundant Error Detector Driver 210 is responsible forconverting all these potentially different formats to one suitable foraction by Redundant Error Detector 230.

In another aspect, RED is integrated into existing analyzers easilybecause it requires few modifications and leaves the subsysteminput-output traffic largely unchanged. This is readily seen from FIGS.1 and 2, wherein Redundant Error Detector 230 is connected withoutsignificantly affecting the inner message traffic of the clinicaldiagnostic analyzer. Preferably, RED can be configure dynamically, forinstance, to release all results. Thus, preferably it is possible tocontrol or configure the penalty imposed by RED when an anomaly isdetected.

Goals for an efficient error detection system include timely detectingerrors to prevent release of erroneous results. The configuration filesdefine the checks performed by RED. Thus, for each assay model type REDpreferably checks for events critical to result quality; eventsdetrimental to result quality; Pass/Fail/Warn criteria, if any, for eachevent; and required sample consistency checks.In most instances, the events checked for by RED are also checked for bythe routine command-response design of the clinical diagnostic analyzeror other system of interest. In view of the small impact of RED onanalyzer performance, implementing RED does not significantly delay therelease of acceptable results.In an effort to reduce the likelihood of RED preventing the predictionof good results helpful guidelines include limiting the number of eventschecked to only the required events; correctly setting the valid rangesof parameters to account for all scenarios; and providingconfigurability to prevent the RED module from being able to failresults even when it is suspected to be malfunctioning.Further, the RED module itself is tested to verify all checks areexecuting correctly in the foreseen scenarios. This may be performed by,for instance, the simulation strategy presented in FIG. 2.

Alternative Error Detection Strategies

RED detects and prevents potentially misreported results. It iscustomary to primarily rely upon testing and code reviews as thepreferred approaches for achieving a desired quality level. However, REDgreatly reduces the need for code reviews, which are necessarilydifficult and resource intensive exercises when testing complicatedmodules.Closed Loop Checking is another strategy that incurs significantdevelopment, integration, and verification expenses. Further, closedloop checking performed within the module in question may not bereliable in all situations. Closed loop checking performed within amodule is often not independent of the module being tested.The RED architecture can accept many different types of inputs. The twoobvious choices are Subsystem I/O (composed of Subsystem Commands,Responses, and Asynchronous messages) and DataLogger events.In a preferred embodiment RED performs the requested checks and passesback a response indicating the status (pass, fail, or optionally—awarning level). RED recognizes that each assay model will have differentprotocol steps and therefore different integrity checks. The categoriesof checks associated with a replicate are listed. Each categorycorrelates with a table of expected events that may be dynamicallyaltered depending upon replicate protocol information. Thus, REDpotentially simplifies coding for the analyzer and provides improvedreliability to the results by providing a test independent of thecomplexity inherent in real-time handling of message traffic in ananalyzer or another complex system. FIG. 5 illustrates the REDfunctionality. Typically, there is a mechanism for transferring messagesand information, here RED Buffer 500, which provides commands and datato Redundant Error Detector Functionality 510. Redundant Error DetectorFunctionality 510 includes, in a preferred embodiment, a configurationfile, fingerprint and its generation capability, capability to comparefingerprint to messages from RED Buffer 500, result reporting capabilityand resource allocation and release functions. The Assay Database 520 isaccessible to Redundant Error Detector Functionality 510, which accessis useful for generating the fingerprints.

One skilled in the art will appreciate this disclosure is susceptible tomany variations and alternative implementations without departing fromits teachings or spirit. For instance, complex systems other thanclinical analyzers will benefit by redundant error testing as describedherein based on evaluating each output for consistency evaluated using afingerprint of events. The scope of the claims appended below includesmany such modifications. Further, each reference discussed and citedherein is hereby incorporated herein by reference in its entirety.

1. A module executable on a processor in a clinical diagnostic analyzerfor detecting questionable laboratory results in the clinical diagnosticanalyzer, the module comprising: a mechanism for receiving messagescomprising Subsystem Commands, Command Responses, and Asynchronousmessages; a fingerprint of expected events to be tested for a replicate,wherein the fingerprint is generated from at least a configurationinformation and protocol information, wherein the protocol informationcontains details for an assay required by the replicate; instructionsfor completing a comparison between the received messages with thefingerprint; and instructions for reporting a result of the comparison.2. The module of claim 1 further comprising instructions for generatingthe fingerprint in response to receiving a Preview command for thereplicate.
 3. The module of claim 1 further comprising instructions forcompleting, in response to a Check command, a comparison of thefingerprint with received messages.
 4. The module of claim 1 comprisinginstructions for releasing all resources, which were sequestered inresponse to receiving the Preview command for the replicate, in responseto receiving a Complete command.
 5. The module of claim 1 comprisinginstructions for dynamically reconfiguring the instructions forreleasing a result of the comparison.
 6. The module of claim 1comprising instructions for logging conditions in response todetermining the result is suspect.
 7. The module of claim 1 comprisinginstructions for logging conditions in response to flagging the resultsof the assay with a warning.
 8. The module of claim 1 further comprisinginstructions for generating events corresponding to a subsystem inresponse to receiving a Preview command for the replicate, wherein thegeneration is based on the configuration information.
 9. A qualitycontrol method for testing a software module in a clinical diagnosticanalyzer comprising the steps of: generating subsystem commands, commandresponses, and asynchronous messages suitable for delivery to aredundant error detector; reviewing the generated subsystem commands,command responses, and asynchronous messages using the redundant errordetector for inconsistencies; and debug the software module in responseto detecting inconsistencies.
 10. The method of claim 9, wherein thesubsystem commands, command responses, and asynchronous messages aregenerated in a simulation of a performance of the clinical diagnosticanalyzer using at least one script file which contains at least onecommand in addition to a record of messages in an example operation ofthe clinical diagnostic analyzer.
 11. The method of claim 10, whereinthe software module is used in the example operation of the clinicaldiagnostic analyzer to generate the record of messages.
 12. The methodof claim 9, wherein the redundant error detector uses a configurationfile and an assay database to generate a fingerprint for at least onereplicate in the at least one script file.
 13. The method of claim 9,wherein the software module is a test redundant error detector.
 14. Themethod of claim 9, wherein the software module is not a test redundanterror detector.
 15. The method of claim 9, wherein a system is tested bytesting several software modules.
 16. A method for preventing release ofa questionable result by a clinical diagnostic analyzer, the methodcomprising facilitating the steps of: receiving a preview replicatecommand for a replicate; allocating resources for redundant errorchecking for the replicate; generating a fingerprint of required eventsfor the replicate using at least a configuration information and anprotocol information, wherein the fingerprint has fewer events than theevents performed in accordance with the protocol information for thereplicate; comparing messages corresponding to carrying out thereplicate with the fingerprint; sending a result of the comparison; andlogging conditions corresponding to the replicate.
 17. The method ofclaim 16, wherein the configuration information is coded in eXtensibleMarkup Language.
 18. The method of claim 16, wherein logging conditionsincludes writing to a log file in a specified format data relating tospecified events.
 19. The method of claim 16, wherein the resultcorresponding to the replicate being fail causes the clinical diagnosticanalyzer to not release the result.